Deep Learning from Scratch in C++: Tensor Programming

Luiz doleron

·

Follow

Published in

Artificial Intelligence in Plain English

·

12 min read

·

May 30

90

1

Let’s have fun by learning how to use the Eigen Tensor API.

Tensors are the primary way to represent data in Deep Learning algorithms. They are widely used to implement inputs, outputs, parameters, and internal states during the algorithm execution.

In this story, we are going to learn how to use the Eigen Tensor API to develop our C++ algorithms. Specifically, we shall talk about:

• What are tensors
• How to define tensors in C++
• How to compute tensor operations
• Tensor Reductions and Convolutions

At the end of this story, we shall implement Softmax as an illustrative example of applying tensors for deep learning algorithms.

Text for beginners: this text requires entry-level programming background and a basic understanding of machine learning.

What is a Tensor?

Tensors are grid-like data structures that generalize the concepts of vectors and matrices for an arbitrary number of axis. In machine learning, we usually use the word “dimension” instead of “axis”. The number of different dimensions of a tensor is also called tensor rank:

In practice, we use tensors to represent data in our algorithms, performing arithmetic operations with them.

The simpler operation we can perform with tensors are the so-called element-wise operations: given two operand tensors with the same dimensions, the operation results in a new tensor with the same dimensions where the value of each coefficient is obtained from the binary evaluation on the respective elements in the operands:

The example above is an illustration of the coefficient-wise product of two 2-rank tensors. This operation is still valid for any two tensors since they have the same dimensions.

Like matrices, we can perform other more sophisticated operations with tensors, such as the matrix-like product, convolutions, contractions, reductions, and a myriad of geometric operations. In this story, we will learn how to use the Eigen Tensor API to perform some of these tensor operations, focusing on the most important for implementing deep learning algorithms.

How to declare and use tensors in C++

As we know, Eigen is a linear algebra library broadly used for matrix computations. In addition to the well know support for matrices, Eigen also has an (unsupported) module for Tensors.

Although the Eigen Tensor API is denoted as unsupported, it is actually well supported by the developers of the Google TensorFlow framework.

We can easily define a tensor using Eigen:

#include <iostream>

#include <unsupported/Eigen/CXX11/Tensor>

int main(int, char **)
{

    Eigen::Tensor<int, 3> my_tensor(2, 3, 4);
    my_tensor.setConstant(42);

    std::cout << "my_tensor:\n\n" 
              << my_tensor << "\n\n";

    std::cout << "tensor size is " << my_tensor.size() << "\n\n"; 

    return 0;
}

The line

Eigen::Tensor<int, 3> my_tensor(2, 3, 4);

creates a tensor object and allocates the required memory to store 2x3x4 ints. In this example,my_tensor is a 3-rank tensor where the size of the first dimension is 2, the size of the second dimension is 3, and the size of the last dimension is 4. We can represent my_tensor as follows:

We can set the tensor data if we want it:

my_tensor.setValues({{{1, 2, 3, 4}, {5, 6, 7, 8}}});

std::cout << "my_tensor:\n\n" << my_tensor << "\n\n";

or use random values instead. For example, we can do:

Eigen::Tensor<float, 2> kernel(3, 3);
kernel.setRandom();
std::cout << "kernel:\n\n" << kernel << "\n\n";

and use this kernel later on to perform convolutions. We will cover convolutions soon in this story. First, let’s learn how to use TensorMaps.

Creating tensor views with Eigen::TensorMap

Sometimes, we have some data allocated and only want to manipulate it using a tensor. Eigen::TensorMap is similar to Eigen::Tensor but, instead of allocating new data, it is only a view of the data passed as the parameter. Check the example below:

//an vector with size 12
std::vector<float> storage(4*3);

// filling vector from 1 to 12
std::iota(storage.begin(), storage.end(), 1.);

for (float v: storage) std::cout << v << ','; 
std::cout << "\n\n";

// setting a tensor view with 4 rows and 3 columns
Eigen::TensorMap<Eigen::Tensor<float, 2>> my_tensor_view(storage.data(), 4, 3);

std::cout << "my_tensor_view before update:\n\n" << my_tensor_view << "\n\n";

// updating the vector
storage[4] = -1.;

std::cout << "my_tensor_view after update:\n\n" << my_tensor_view << "\n\n";

// updating the tensor
my_tensor_view(2, 1) = -8;

std::cout << "vector after two updates:\n\n";
for (float v: storage) std::cout << v << ','; 
std::cout << "\n\n";

In this example, it is easy to see that (by default) Tensors in Eigen Tensor API are col-major. col-major and row-major refer to the way how the grid data is stored in linear containers (check this article on Wikipedia):

Although we can use row-major tensors, it is not recommended:

Only the default column major layout is currently fully supported, and it is therefore not recommended to attempt to use the row major layout at the moment.

Eigen::TensorMap is very useful because we can use it to save memory, which is critical for high-demanding applications such as deep learning algorithms.

Performing unary and binary operations

The Eigen Tensor API defines common arithmetic overload operators, making programming tensors intuitive and straightforward. For example, we can add and subtract tensors:

Eigen::Tensor<float, 2> A(2, 3), B(2, 3);
A.setRandom();
B.setRandom();

Eigen::Tensor<float, 2> C = 2.f*A + B.exp();

std::cout << "A is\n\n"<< A << "\n\n";
std::cout << "B is\n\n"<< B << "\n\n";
std::cout << "C is\n\n"<< C << "\n\n";

The Eigen Tensor API has several other element-wise functions like .exp() such as sqrt() , log() , and abs() . In addition, we can use unaryExpr(fun) as follows:

auto cosine = [](float v) {return cos(v);};

Eigen::Tensor<float, 2> D = A.unaryExpr(cosine);

std::cout << "D is\n\n"<< D << "\n\n";

Similarly, we can use binaryExpr :

auto fun = [](float a, float b) {return 2.*a + b;};

Eigen::Tensor<float, 2> E = A.binaryExpr(B, fun);

std::cout << "E is\n\n"<< E << "\n\n";

Lazy evaluation and the auto keyword

The Google engineers who worked on the Eigen Tensor API followed the same strategies found at the top of the Eigen Library. One of these strategies, and probably the most important one, is the way how expressions are lazily evaluated.

The lazy evaluation strategy consists of retarding the actual evaluation of expressions to combine multiple chained expressions in an optimized equivalent one. Thus, instead of evaluating multiple individual expressions step-by-step, the optimized code evaluates only one expression, aiming to leverage the final overall performance.

For example, if A and B are tensors, the expression A + B does not actually evaluates the sum of A and B. In fact, the expressionA + B results in a special object that knows how to compute A + B . The operation will only be performed when this special object is assigned to an actual tensor. In other words, in the following statement:

auto C = A + B;

C is not the actual result of A + B but just a computation object (an Eigen::TensorCwiseBinaryOp object indeed) that knows how to calculate A + B. Only when C is assigned to a tensor object (an object of typeEigen::Tensor, Eigen::TensorMap, Eigen::TensorRef, etc) it will be evaluated to provide the proper tensor values:

Eigen::Tensor<...> T = C;
std::cout << "T is " << T << "\n\n";

Of course, this does not make sense for small operations like A + B . However, this behavior is useful for long operation chains, where the computation can be optimized before being evaluated. In resume, as a general guideline, instead of writing a code like this:

Eigen::Tensor<...> A = ...;
Eigen::Tensor<...> B = ...;
Eigen::Tensor<...> C = B * 0.5f;
Eigen::Tensor<...> D = A + C;
Eigen::Tensor<...> E = D.sqrt();

we should write a code like this:

Eigen::Tensor<...> A = ...;
Eigen::Tensor<...> B = ...;
auto C = B * 0.5f;
auto D = A + C;
Eigen::Tensor<...> E = D.sqrt();

The difference is that, in the former, C and D are actually Eigen::Tensor objects, whereas in the later code, they are only lazy-computation operations.

In resume, using lazy computations to evaluate a long chain of operations is preferable because the chain will be internally optimized, eventually resulting in faster executions.

Geometric Operations

Geometrics operations result in tensors with different dimensions and, sometimes, sizes. Examples of these operations are: reshape , pad , shuffle , stride , and broadcast .

It is noteworthy that the Eigen Tensor API does not have a transpose operation. We can emulate transpose using shuffle though:

auto transpose(const Eigen::Tensor<float, 2> &tensor) {
    Eigen::array<int, 2> dims({1, 0});
    return tensor.shuffle(dims);
}

Eigen::Tensor<float, 2> a_tensor(3, 4);
a_tensor.setRandom();

std::cout << "a_tensor is\n\n"<< a_tensor << "\n\n";
std::cout << "a_tensor transpose is\n\n"<< transpose(a_tensor) << "\n\n";

We will see some examples of geometric operations later on when we talk about the softmax example using tensors.

Reductions

Reductions are a special case of operations that results in a tensor with fewer dimensions than the original. Intuitive cases of reductions are sum() and maximum() :

Eigen::Tensor<float, 3> X(5, 2, 3);
X.setRandom();

std::cout << "X is\n\n"<< X << "\n\n";

std::cout << "X.sum(): " << X.sum() << "\n\n";
std::cout << "X.maximum(): " << X.maximum() << "\n\n";

In the example above, we reduced all the dimensions once. We can also perform reductions along specific axes. For example:

Eigen::array<int, 2> dims({1, 2});

std::cout << "X.sum(dims): " << X.sum(dims) << "\n\n";
std::cout << "X.maximum(dims): " << X.maximum(dims) << "\n\n";

The Eigen Tensor API has a set of pre-built reducing operations, such as prod , any , all , mean , and others. If any of the pre-built operations are not suitable for a specific implementation, we can use reduce(dims, reducer) providing a customreducer functor as a parameter.

Tensor Convolutions

In one of the previous stories, we learned how to implement 2D convolutions using only plain C++ and Eigen Matrices. Indeed, it was necessary because there is no built-in convolution for matrices in Eigen. Luckily, the Eigen Tensor API has a handy function to perform convolutions on Eigen Tensor objects:

Eigen::Tensor<float, 4> input(1, 6, 6, 3);
input.setRandom();

Eigen::Tensor<float, 2> kernel(3, 3);
kernel.setRandom();

Eigen::Tensor<float, 4> output(1, 4, 4, 3);

Eigen::array<int, 2> dims({1, 2});
output = input.convolve(kernel, dims);

std::cout << "input:\n\n" << input << "\n\n";
std::cout << "kernel:\n\n" << kernel << "\n\n";
std::cout << "output:\n\n" << output << "\n\n";

Note that we can perform 2D, 3D, 4D, etc, convolutions by controlling the dimensions of the slide in the convolution.

Softmax with tensors

When programming deep learning models, we use tensors instead of using matrices. It turns out that matrices can represent one or at most two-dimensional grids meanwhile we have higher-dimensional data multi-channel images or batches of registers to handle. That is where tensors come to play.

Let’s consider the following example where we have two batches of registers, each batch having four registers and each register having three values:

We can represent this data as follows:

Eigen::Tensor<float, 3> input(2, 4, 3);
input.setValues({
    {{0.1, 1., -2.},{10., 2., 5.},{5., -5., 0.},{2., 3., 2.}},
    {{100., 1000., -500.},{3., 3., 3.},{-1, 1., -1.},{-11., -0.2, -.1}}
});

std::cout << "input:\n\n" << input << "\n\n";

Now, let’s apply softmax to this data:

Eigen::Tensor<float, 3> output = softmax(input);
std::cout << "output:\n\n" << output << "\n\n";

Softmax is a popular activation function. We covered its implementation using Eigen::Matrixin a previous story. Now, let’s introduce the implementation using Eigen::Tensor instead:

#include <unsupported/Eigen/CXX11/Tensor>

auto softmax(const Eigen::Tensor<float, 3> &z)
{

    auto dimensions = z.dimensions();

    int batches = dimensions.at(0);
    int instances_per_batch = dimensions.at(1);
    int instance_length = dimensions.at(2);

    Eigen::array<int, 1> depth_dim({2});
    auto z_max = z.maximum(depth_dim);

    Eigen::array<int, 3> reshape_dim({batches, instances_per_batch, 1});
    auto max_reshaped = z_max.reshape(reshape_dim);

    Eigen::array<int, 3> bcast({1, 1, instance_length});
    auto max_values = max_reshaped.broadcast(bcast);

    auto diff = z - max_values;

    auto expo = diff.exp();
    auto expo_sums = expo.sum(depth_dim);
    auto sums_reshaped = expo_sums.reshape(reshape_dim);
    auto sums = sums_reshaped.broadcast(bcast);
    auto result = expo / sums;

    return result;
}

This code outputs:

We will not go into the Softmax details here. Do not hesitate to read again the previous story here on Medium if you need a review of the Softmax algorithm. Now, we are only focused on understanding how to use Eigen Tensors to code our Deep Learning models.

The first thing to note is that the function softmax(z) does not actually calculate the value of softmax for the parameter z . In fact, softmax(z) only mounts a complex object which can compute softmax.

The actual value will be evaluated only when the result of softmax(z) is assigned to a tensor-like object. For example, here:

Eigen::Tensor<float, 3> output = softmax(input);

Before this line, everything is only the computation graph of softmax, hopefully optimized. This happened only because we used the keyword auto in the body of softmax(z) . Thus, the Eigen Tensor API can optimize the whole computation of softmax(z) using fewer operations, which improves both processing and memory usage.

Before finishing this story, I would like to point out the tensor.reshape(dims) and tensor.broadcast(bcast) calls:

Eigen::array<int, 3> reshape_dim({batches, instances_per_batch, 1});
auto max_reshaped = z_max.reshape(reshape_dim);

Eigen::array<int, 3> bcast({1, 1, instance_length});
auto max_values = max_reshaped.broadcast(bcast);

reshape(dims) is a special geometric operation that generates another tensor with the same size as the original tensor but with different dimensions. Reshape does not change the order of the data internally in the tensor. For example:

Eigen::Tensor<float, 2> X(2, 3);
X.setValues({{1,2,3},{4,5,6}});

std::cout << "X is\n\n"<< X << "\n\n";

std::cout << "Size of X is "<< X.size() << "\n\n";

Eigen::array<int, 3> new_dims({3,1,2});
Eigen::Tensor<float, 3> Y = X.reshape(new_dims);

std::cout << "Y is\n\n"<< Y << "\n\n";

std::cout << "Size of Y is "<< Y.size() << "\n\n";

Note that, in this example, the size of X and Y is either 6, although they have very different geometry.

tensor.broadcast(bcast) repeats the tensor as many times as provided in the bcast parameter for each dimension. For example:

Eigen::Tensor<float, 2> Z(1,3);
Z.setValues({{1,2,3}});
Eigen::array<int, 2> bcast({4, 2});
Eigen::Tensor<float, 2> W = Z.broadcast(bcast);

std::cout << "Z is\n\n"<< Z << "\n\n";
std::cout << "W is\n\n"<< W << "\n\n";

Different of reshape, broadcast does not changes the tensor rank (i.e., the number of dimensions) but only increases the size of the dimensions.

Limitations

The Eigen Tensor API docs cite some limitations of which we can be aware:

• The GPU support was tested and optimized to the float type. Even if we can declare Eigen::Tensor<int,...> tensor; , the usage of non-float tensors is discouraged when using the GPU.
• The default layout (col-major) is the only one actually supported. We shouldn’t use row-major, at least for now.
• The max number of dimensions is 250. This size is only achieved when using a C++11-compatible compiler.

Conclusion and Next Steps

Tensors are essential data structures for machine learning programming, allowing us to represent and handle multi-dimensional data as straightforwardly as regular two-dimensional matrices.

In this story, we introduced the Eigen Tensor API and learned how to use tensors relatively easily. We also learned that the Eigen Tensor API has a lazy evaluation mechanism, resulting in an optimized execution in terms of memory and processing time.

To ensure that we understood Eigen Tensor API usage, we covered an example of coding Softmax using tensors.

In the next stories, we will continue developing high-performing deep learning algorithms from scratch using C++ and Eigen, in particular, using the Eigen Tensor API.

Code

You can find the code used in this story in this repository on GitHub.

References

[1] Eigen Tensor API

[2] Eigen Tensor Module

[3] Eigen Gitlab repository, https://gitlab.com/libeigen/eigen

[4] Charu C. Aggarwal, Neural Networks and Deep Learning: A Textbook (2018), Springer

[5] Jason Brownlee, A Gentle Introduction to Tensors for Machine Learning with NumPy

About this series

In this series, we will learn how to code the must-to-know deep learning algorithms such as convolutions, backpropagation, activation functions, optimizers, deep neural networks, and so on using only plain and modern C++.

This story is: Using Eigen Tensor API

Check other stories:

0 — Fundamentals of deep learning programming in Modern C++

1 — Coding 2D convolutions in pure C++

2 — Cost Functions using Lambdas

3 — Implementing Gradient Descent

4 — Activation Functions

… more to come.

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.